Hybrid see through augmented reality systems and methods for low vision users

ABSTRACT

Provided herein are augmented reality visual aid systems, software, and methods which enhance vision, to simulate natural vision, by utilizing hybrid see through occlusion enabled hardware, and software through image manipulation, reprocessing, blending, for presentation and display to the eyes thus enabling a range of tasks previously lost or impacted.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/537,193, filed Nov. 29, 2021, now U.S. Pat. No. 11,385,468, which isa continuation of U.S. application Ser. No. 17/057,181, filed Nov. 20,2020, now U.S. Pat. No. 11,187,906, which application is the U.S.national phase of International Application No. PCT/US2019/034443, filedMay 29, 2019, which claims benefit of priority to U.S. ProvisionalPatent Application No. 62/677,463, filed May 29, 2018, titled “HybridSee Through Augmented Reality Suitable for Low Vision Users,” hereinincorporated by reference in their entirety.

This application is related to U.S. Appln. No. 62/530,286, filed Jul. 9,2017, U.S. Appln. No. 62/530,792, filed Jul. 10, 2017, U.S. Appln. No.62/579,657, filed Oct. 31, 2017, U.S. Appln. No. 62/579,798, filed Oct.31, 2017, International Appln. No. PCT/US17/062421, filed Nov. 17, 2017,U.S. application Ser. No. 15/817,117, filed Nov. 17, 2017, U.S. Appln.No. 62/639,347, filed Mar. 6, 2018, U.S. application Ser. No.15/918,884, filed Mar. 12, 2018, U.S. Appln. No. 62/735,643, filed Sep.24, 2018, and U.S. Appln. No. 16,177,333, filed Oct. 31, 2018, thecontents of which are incorporated herein by reference herein in theirentirety.

INCORPORATION BY REFERENCE

All publications and patent applications mentioned in this specificationare herein incorporated by reference in their entirety to the sameextent as if each individual publication or patent application wasspecifically and individually indicated to be incorporated by reference.

BACKGROUND

Head mounted visual aids have been used for hundreds of years, and as inthe past are commonly optics-based solutions such as eyeglasses. Theconceptualization and early experimentation with programmable headmounted electronic based visual aids began with NASA-funded research inthe late 1980s. Basic functionality described included remapping ofpixels in order to manipulate the image presented to the wearer's eye.The remapping described was primarily forms of magnification and warpingof the image with different shapes, sizes and positions of the modifiedarea. Although early experiments proved noteworthy for several retinalbased diseases, ultimately the research proved impractical at the timefor a variety of reasons.

Current hardware implementations of wearable head mounted devices havebecome more common, with Virtual Reality (VR) headsets and AugmentedReality (AR) glasses becoming mainstream. These platforms were designedand intended for use in a number of applications such as gaming,telepresence and a wide variety of enterprise applications. Thisenabling technology also provides a potential fruitful basis for moreadvanced image processing for the low vision user. However, neitherapproach is ideal for the low vision user, and requires furtherrefinement along with novel software and algorithms to work seamlesslytogether to make the head mounted wearable practical and useful for thelow vision user.

SUMMARY

The inventions described herein relate to improved hardware andintegrated software and algorithms for a class of wearable electronicAugmented Reality (AR) glasses in general, with additional specificbenefits for low-vision users suffering from various visual impairments(e.g., Age-related Macular Degeneration—AMD—and other visual fielddeficiencies). The presented adjustments to pre-existing AR glassdesigns resolve a problem previously addressed by brute-force methodsthat ultimately reduce the usability and versatility of the glasses. Byinstead acknowledging and understanding the underlying phenomenology,the updated approach establishes a new AR paradigm that further enhancesthe experience of low-vision users without sacrificing key benefits ofthe standard design. In certain applications, normally-sighted userswill also benefit from these changes.

Present Augmented Reality (AR) eyewear implementations fall cleanly intotwo disjoint categories, video see-through (VST) and optical see-through(OST).

Apparatuses for VST AR closely resembles Virtual Reality (VR) gear,where the wearer's eyes are fully enclosed so that only content directlyshown on the embedded display is visible. VR systems maintain afully-synthetic three-dimensional environment that must be continuouslyupdated and rendered at tremendous computational cost. VST AR glassesalso fully enclose the eyes, but instead present imagery based on thereal-time video feed from an appropriately-mounted camera (or cameras)directed along the user's eyeline; hence the data and problem domain arefundamentally two-dimensional. Like VR, VST AR provides absolute controlover the final appearance of visual stimulus, and facilitatesregistration and synchronization of captured video with any syntheticaugmentations. Very wide fields-of-view (FOV) approximating naturalhuman limits are also achievable at low cost. However, VST gear tends tobe bulky and incur additional latencies associated with image capture.Furthermore, complete immersion of the user also results of loss ofperipheral vision and hence mobility that requires peripheral vision; itis also commonly associated with side effects such as dizziness andnausea caused by unanchoring from reality. VST in the VR domain also canhave the drawback of a sharp demarcation if the field of view is notsufficient to cover the entire area of interest for the user.

OST AR eyewear, on the other hand, has a direct optical path allowinglight from the scene to form a natural image on the retina. This naturalimage is essentially the same one that would be formed without ARglasses, possibly with some loss of brightness due to attenuation byoptical components. A camera is used to capture the scene for automatedanalysis, but its image does not need to be shown to the user. Instead,computed annotations or drawings from an internal display aresuperimposed onto the natural retinal image by any of a number ofestablished optical combining methods, (e.g.) direct laser projectiononto the retina, electronic contact lenses, birdbath optics, or ahalf-silvered mirror for optical combining. In a traditional OST ARapplication, the majority of the display typically remains blank (i.e.black) to avoid contributing any photons to the final retinal image;displayed augmentations produce sufficient light to be visible againstthe natural image background. The horizontal field-of-view over whichannotations can be projected tends to be limited to a central 25 to 50degrees, but there is no delay between real-world events and theirperception. Furthermore, the scene image has no artifacts due toimage-sensor sampling, capture, or processing. However, synchronizingaugmentations becomes more challenging and user-dependent calibrationmay be needed to ensure proper their registration. Finally, OSTpossesses an inherent degree of safety that VST lacks: if the OSThardware fails, the user can still see the environment. However, directOST is often confusing for low vision users due to the difficulty ofresolving the double image resulting from the AR overlay.

The primary task of visual-assistance eyewear for low-vision sufferersdoes not match the most common use model for AR (whether VST or OST),which involves superimposing annotations or drawings on a backgroundimage that is otherwise faithful to the objective reality that would beseen by the unaided eye. Instead, assistive devices must dramaticallychange how the environment is displayed in order to compensate defectsin the user's vision. Processing may include such effects as contrastenhancement and color mapping, but invariably incorporates increasedmagnification to counteract deficient visual acuity. Existing devicesfor low-vision are magnification-centric, and hence operate in the VSTregime with VST hardware. Some use OST-based AR platforms, but installopaque lens covers that completely block all environmental light fromentering the retina—since a camera supplies the only visible image viaan internal display, they become exclusively VST systems.

The inventions described here instead employ a unique combined VST/OSTmethodology (hybrid see-through, or HST) to produce a final retinalimage. Doing so permits the best characteristics of each technique to beeffectively exploited while simultaneously avoiding or amelioratingundesirable aspects. Specifically:

The wide field of view associated with VST can be maintained for theuser in spite of the narrow active display area of the OST-based image;

Absolute control over the final retinal image details is achieved (as inVST) for the highest-acuity central area covered by the internaldisplay;

Additional blending of the augmented image and the real world can beaccomplished through software that performs non-linear transformationsand image remapping to avoid harsh and abrupt transitions between theprocessed image and unmodified reality

A fail-safe vision path exists at all times (as in OST), regardless ofthe content of the internal display—and whether or not that display isfunctioning;

Difficulty of resolving focus ambiguities between natural images and OSToverlays in low-vision users is addressed and remedied.

A hybrid see-through augmented reality device is provided, comprising aframe configured to be worn on the head of a user, a camera disposed onthe frame and configured to generate unprocessed real-time video images,a first display disposed within the frame and including a first barrierconfigured to substantially prevent external light corresponding to acentral portion of the user's field of view from entering a first eye ofthe user while allowing external light corresponding to a peripheralportion of the user's field of view to enter the first eye of the user,a processor disposed on or in the frame and configured to process thereal-time video images from the camera to produce a video stream that,when displayed on the first display, replaces the central portion of theuser's field of view while blending with the peripheral portion of theuser's field of view.

In some aspects, the device further includes a second display disposedwithin the frame and including a second barrier configured tosubstantially prevent external light corresponding to the centralportion of the user's field of view from entering a second eye of theuser while allowing external light corresponding to the peripheralportion of the user's field of view to enter the second eye of the user.

In one example, the processor is also configured to display the videostream on the second display.

In another example, the processor is configured to process theunprocessed real-time video images to produce the video stream that isenhanced when compared to the unprocessed real-time video images. Theenhanced video stream can be at least partially magnified when comparedto the unprocessed real-time video images. In some examples, the videostream is magnified in a central portion of the video stream. In anotherexample, a portion of the video stream outside of the central portion ismagnified less than the central portion but more than the unprocessedreal-time video images.

In one example, the processor is configured to process the unprocessedreal-time video images to blend a top edge, a bottom edge, a left edge,and a right edge of the video stream with the peripheral portion of theuser's field of view. In another example, the processor is configured toprocess the unprocessed real-time video images to blend only a left edgeand a right edge of the video stream with the peripheral portion of theuser's field of view.

In some examples, the processor is configured to process the unprocessedreal-time video with image coordinate remapping to blend the videostream with the peripheral portion of the user's field of view. In someexamples, the image coordinate remapping comprises radial mapping.

In one example, the device further comprises an input device configuredto receive an input from the user regarding a type and/or an amount ofenhancement to apply to the unprocessed real-time video images. In someexamples, the input device comprises a physical mechanism. In otherexamples, the input device comprises a microphone disposed on or in thehousing configured to receive voice commands from the user.

A method of providing enhanced vision for a low-vision user is alsoprovided, comprising generating unprocessed real-time video images witha head mounted camera, preventing light corresponding to a centralportion of the user's field of view from entering a first eye of theuser, processing the unprocessed real-time video images to produce avideo stream that corresponds to the central portion of the user's fieldof view and blends with a peripheral portion of the user's field ofview, and displaying the video stream on a display positioned within thecentral portion of the user's field of view that substantially preventsexternal light corresponding to the central portion of the user's fieldof view from entering the first eye of the user while allowing externallight corresponding to the peripheral portion of the user's field ofview to enter the first eye of the user.

In some examples, the method further comprises displaying the videostream on a second display positioned within the central portion of theuser's field of view. In one example, the display is positioned in frontof the first eye of the user and the second display is positioned infront of a second eye of the user.

In other examples, the method further comprises processing theunprocessed real-time video images such that the video stream isenhanced when compared to the unprocessed real-time video images. In oneexample, the enhanced video stream is at least partially magnified whencompared to the unprocessed real-time video images. In another example,the enhanced video stream is magnified in a central portion of the videostream. In one example, a portion of the video stream outside of thecentral portion is magnified less than the central portion but more thanthe unprocessed real-time video images.

In some examples the method further comprises processing the unprocessedreal-time video images to blend a top edge, a bottom edge, a left edge,and a right edge of the video stream with the peripheral portion of theuser's field of view. In other examples the method further comprisesprocessing the unprocessed real-time video images to blend only a leftedge and a right edge of the video stream with the peripheral portion ofthe user's field of view.

In some examples, the method further comprises processing theunprocessed real-time video with image coordinate remapping to blend thevideo stream with the peripheral portion of the user's field of view. Inone example, the image coordinate remapping comprises radial mapping.

A non-transitory computing device readable medium is also provided, themedium having instructions stored thereon that are executable by aprocessor to cause a computing device to generate unprocessed real-timevideo images, process the unprocessed real-time video images to producea video stream that corresponds to a central portion of a user's fieldof view and blends seamlessly with a peripheral portion of the user'sfield of view, display the video stream on a display positioned withinthe central portion of the user's field of view that substantiallyprevents external light corresponding to the central portion of theuser's field of view from entering a first eye of the user whileallowing external light corresponding to the peripheral portion of theuser's field of view to enter the first eye of the user.

In some examples, the instructions further cause the computing device todisplay the video stream on a second display positioned within thecentral portion of the user's field of view.

In one example, the display is positioned in front of the first eye ofthe user and the second display is positioned in front of a second eyeof the user.

In other examples, the instructions further cause the computing deviceto process the unprocessed real-time video images such that the videostream is enhanced when compared to the unprocessed real-time videoimages. In one example, the enhanced video stream is at least partiallymagnified when compared to the unprocessed real-time video images. Inanother example, the enhanced video stream is magnified in a centralportion of the video stream.

In one example, a portion of the video stream outside of the centralportion is magnified less than the central portion but more than theunprocessed real-time video images.

In some examples, the instructions further cause the computing device toprocess the unprocessed real-time video images to blend a top edge, abottom edge, a left edge, and a right edge of the video stream with theperipheral portion of the user's field of view. In another example, theinstructions further cause the computing device to process theunprocessed real-time video images to blend only a left edge and a rightedge of the video stream with the peripheral portion of the user's fieldof view.

In some examples, the instructions further cause the computing device toprocess the unprocessed real-time video with image coordinate remappingto blend the video stream with the peripheral portion of the user'sfield of view. In one example, the image coordinate remapping comprisesradial mapping.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe claims that follow. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1A illustrates the three types of light rays considered in theformulation of the Hybrid See Through (HST) Augmented Reality (AR)methodology, depicting them in the context of a simple eyeglass-likeOptical See Through (OST) AR wearable platform as viewed from the side.

FIG. 1B illustrates the OST AR wearable platform of FIG. 1A, as viewedfrom the front (i.e. looking directly at the wearer of the OST ARwearable platform).

FIG. 2A is one example of a HST device according to the presentdisclosure.

FIG. 2B is one example of a HST device according to the presentdisclosure.

FIG. 2C is one example of a HST device according to the presentdisclosure.

FIG. 2D is one example of a HST device according to the presentdisclosure.

FIG. 2E is one example of a HST device, including some of the detailedcomponents, according to the present disclosure.

FIG. 3A shows the outer (scene-facing) side of a lens modified with anopaque barrier to support HST for an OST device.

FIG. 3B shows the inner (eye-facing) side of the lens of FIG. 3A.

FIG. 4A contains a simulated view of a HST device with no processingexcept for a rectangle showing the picture of a display with 50-degreediagonal field-of-view

FIG. 4B shows the same view as FIG. 4A, but simulates processing thatincludes central magnification with seamless transitions between imagesat the edge of the display and objects outside of the display.

FIG. 4C shows the same view as FIGS. 4A and 4B, but simulates processingthat includes central magnification, with seamless transitions betweenimages at the left and right edges of the display and objects outsidethe display, but only perceptually-seamless transitions between imagesat the upper and lower edge of the display and objects outside of thedisplay.

FIG. 5A is a high-level dataflow diagram showing how data, controls, andstate interact with the main components of a HST device to performeffective low-vision processing.

FIG. 5B is a high-level flowchart showing the most significant paralleland sequential steps that occur in a HST device in order to performeffective low-vision processing.

FIG. 6A (not to scale) demonstrates how a normal human eye can focus theimage from an embedded AR display onto the retina using distance visionwhile viewing a near-field object at typical reading distance of 16-24inches.

FIG. 6B (not to scale) demonstrates how a normal human eye can focus theimage of a near-field object onto the retina using near vision whilewearing a HST device that simultaneously display a far-field object atlarger distances.

FIG. 7A depicts the graph of a radially symmetric mapping function wherethe mapping is the identity transformation.

FIG. 7B depicts the graph of a radially symmetric mapping function wherethe mapping provides a uniform magnification of 2×.

FIG. 7C depicts the graph of a radially symmetric mapping functioncorresponding to the perfect seamlessly-blended HST mapping illustratedin FIG. 4 b , with uniform central magnification of 2× tapering to nomagnification at the periphery.

FIG. 7D depicts the graph of a radially symmetric mapping functionproviding a non-uniform nonlinear magnification profile over the centralvisual field, with seamless blending suitable for a HST device.

DETAILED DESCRIPTION

The present disclosure is related to systems, methods, computing devicereadable media, and devices for providing enhanced vision to persons,users, or patients with low vision, particularly low vision in a centerof the user's field of view (FOV).

For people with retinal diseases, adapting to loss of vision becomes away of life. This impacts their lives in many ways including loss of theability to read, loss of income, loss of mobility and an overalldegraded quality of life. However, with prevalent retinal diseases suchas AMD (Age-related Macular Degeneration) not all of the vision is lost,and in this case the peripheral vision remains intact as only thecentral vision is impacted by the degradation of the macula. Given thatthe peripheral vision remains intact it is possible to take advantage ofeccentric viewing by enhancing and optimizing the peripheral visionwhile perceptually maintaining the FOV which otherwise decreases withincreased magnification. By way of example these disease states may takethe form of age-related macular degeneration, retinitis pigmentosa,diabetic retinopathy, Stargardt's disease, and other diseases wheredamage to part of the retina impairs vision. The present disclosuredescribed herein is novel because it not only supplies systems andmethods to enhance vision, but also provides simple but powerfulhardware enhancements that work in conjunction with the software toprovide a more natural field of view in conjunction with augmentedimages.

Hybrid see through (HST) devices as described herein can be constructedfrom a non-invasive, wearable electronics-based AR eyeglass system (seeFIGS. 2A-2E) employing any of a variety of integrated displaytechnologies, including LCD, OLED, or direct retinal projection.Materials are also able to be substituted for the “glass” havingelectronic elements embedded within the same, so that “glasses” may beunderstood to encompass for example, sheets of lens and cameracontaining materials, IOLs, contact lenses and the like functionalunits. These displays are placed in front of the eyes so as to readilydisplay or project a modified or augmented image when observed with theeyes. This is commonly implemented as a display for each eye, but mayalso work for only one display as well as a continuous large displayviewable by both eyes.

Referring now to FIGS. 2A-2E, HST device 99 is housed in a glasses framemodel including both features and zones of placement which areinterchangeable for processor 101, charging and dataport 103, dualdisplay 111, control buttons 106, accelerometer gyroscope magnetometer112, Bluetooth/Wi-Fi 108, autofocus camera 113, flashlight 125, andspeaker/microphone combinations 120, known to those skilled in the art.For example, batteries 107, including lithium-ion batteries shown in afigure, or any known or developed other versions, functioning as abattery. Power management circuitry is contained within or interfaceswith or monitors the battery to manage power consumption, controlbattery charging, and provide supply voltages to the various deviceswhich may require different power requirements.

As shown in FIGS. 2A-2E, any basic hardware can be constructed from anon-invasive, wearable electronics-based AR eyeglass system (see FIGS.2A-2E) employing any of a variety of integrated display technologies,including LCD, OLED, or direct retinal projection. Materials are alsoable to be substituted for the “glass” having electronic elementsembedded within the same, so that “glasses” may be understood toencompass for example, sheets of lens and camera containing materials,IOLs, contact lenses and the like functional units.

One or more cameras (still, video, or both) 113, mounted on or withinthe glasses, are configured to continuously monitor the view where theglasses are pointing and continuously capture images that are stored,manipulated and used interactively in the HST device. In addition, oneor more of these cameras may be IR (infrared) cameras for observationand monitoring in a variety of lighting conditions. The HST device canalso contain an integrated processor or controller and memory storage(either embedded in the glasses, or tethered by a cable) with embeddedsoftware implementing real-time algorithms configured to modify theimages as they are captured by the camera(s). These modified, orcorrected, images are then continuously presented to the eyes of theuser via the displays.

The processes described herein are implemented in a HST deviceconfigured to present an image or a real-time stream of video to theuser. The processes may be implemented in computer programs (also knownas programs, software, software applications or code) include machineinstructions for a programmable processor, and can be implemented in ahigh-level procedural and/or object-oriented programming language,and/or in assembly/machine language, such as machine readable code ormachine executable code that is stored on a memory and executed by aprocessor. Input signals or data is received by the unit from a user,cameras, detectors or any other device. Other kinds of devices can beused to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback (e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. Output is presented to the user in any manner, includinga screen display or headset display. The processor and memory can beintegral components of the HST device shown in FIGS. 2A-2D, or can beseparate components linked to the HST device. Other devices such asmobile platforms with displays (cellular phones, tablets etc.)electronic magnifiers, and electronically enabled contact lens are alsoable to be used.

HST device 99 includes a processor 252A, memory 264A, an input/outputdevice such as a display 254A, a communication interface 266A, and atransceiver 268A, along with other components. The device 99 may also beprovided with a storage device, such as a Microdrive or other device, toprovide additional storage. Each of the components of HST device 99,252A, 264A, 254A, 266A, and 268A, are interconnected using variousbuses, and several of the components may be mounted on a commonmotherboard or in other manners as appropriate.

The processor 252A can execute instructions within the HST device 99,including instructions stored in the memory 264A. The processor may beimplemented as a chipset of chips that include separate and multipleanalog and digital processors. The processor may provide, for example,for coordination of the other components of the device 99, such ascontrol of user interfaces, applications run by device 99, and wirelesscommunication by device 99.

Processor 252A may communicate with a user through control interface258A and display interface 256A coupled to a display 254A. The display254A may be, for example, a TFT LCD (Thin-Film-Transistor Liquid CrystalDisplay) or an OLED (Organic Light Emitting Diode) display, or otherappropriate display technology. The display interface 256A may compriseappropriate circuitry for driving the display 254A to present graphical,video and other information to a user. The control interface 258A mayreceive commands from a user and convert them for submission to theprocessor 252A. In addition, an external interface 262A may be providedin communication with processor 252A, so as to enable near areacommunication of device 99 with other devices. External interface 262Amay provide for example, for wired communication in someimplementations, or for wireless communication in other implementations,and multiple interfaces may also be used.

The memory 264A stores information within the HST device 99. The memory264A can be implemented as one or more of a computer-readable medium ormedia, a volatile memory unit or units, or a non-volatile memory unit orunits. Expansion memory 274A may also be provided and connected todevice 99 through expansion interface 272A, which may include, forexample, a SIMM (Single In Line Memory Module) card interface. Suchexpansion memory 274A may provide extra storage space for device 99, ormay also store applications or other information for HST device 99.Specifically, expansion memory 274A may include instructions to carryout or supplement the processes described above, and may include secureinformation also. Thus, for example, expansion memory 274A may beprovided as a security module for device 99, and may be programmed withinstructions that permit secure use of device 99. In addition, secureapplications may be provided via the SIMM cards, along with additionalinformation, such as placing identifying information on the SIMM card ina non-backable manner. The memory may include, for example, flash memoryand/or NVRAM memory, as discussed below. In one implementation, acomputer program product is tangibly embodied in an information carrier.The computer program product contains instructions that, when executed,perform one or more methods, such as those described above. Theinformation carrier is a computer- or machine-readable medium, such asthe memory 264A, expansion memory 274A, or memory on processor 252A,that may be received, for example, over transceiver 268A or externalinterface 262A.

HST Device 99 may communicate wirelessly through communication interface266A, which may include digital signal processing circuitry wherenecessary. Communication interface 266A may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, GPRS, EDGE, 3G, 4G, 5G,AMPS, FRS, GMRS, citizen band radio, VHF, AM, FM, and wireless USB amongothers. Such communication may occur, for example, throughradio-frequency transceiver 268A. In addition, short-range communicationmay occur, such as using a Bluetooth, Wi-Fi, or other such transceiversuch as wireless LAN, WMAN, broadband fixed access or WiMAX. Inaddition, GPS (Global Positioning System) receiver module 270A mayprovide additional navigation- and location-related wireless data todevice 99, and is capable of receiving and processing signals fromsatellites or other transponders to generate location data regarding thelocation, direction of travel, and speed, which may be used asappropriate by applications running on HST device 99.

HST device 99 may also communicate audibly using audio codec 260A, whichmay receive spoken information from a user and convert it to usabledigital information. Audio codec 260A may likewise generate audiblesound for a user, such as through a speaker, e.g., in a handset ofdevice 99. Such sound may include sound from voice telephone calls, mayinclude recorded sound (e.g., voice messages, music files, etc.) and mayalso include sound generated by applications operating on device 99.Part of the HST device is a speaker and microphone 120. The speaker andmicrophone may be controlled by the processor 252A and are configured toreceive, generate and convert audio signals to electrical signals, inthe case of the microphone, based on processor control.

An IMU (inertial measurement unit) 280A connects to the bus, or isintegrated with other components, generates and provides data regardingthe orientation of the HST device 99. This IMU can contain a compass,such as a magnetometer, an accelerometer and/or gyro, to providedirectional data, impact and shock data or other information or dataregarding shocks or forces experienced by the HST device.

A flasher and/or flashlight 125 are provided and are processorcontrollable. The flasher or flashlight may serve as a strobe ortraditional flashlight, and may include an LED.

Thus, various implementations of the system and techniques describedhere can be realized in digital electronic circuitry, integratedcircuitry, specially designed ASICs (application specific integratedcircuits), computer hardware, firmware, software, and/or combinationsthereof. These various implementations can include implementation in oneor more computer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

As used herein, the terms “machine-readable medium” “computer-readablemedium” refers to any computer program product, apparatus and/or device(e.g., magnetic discs, optical disks, memory, Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor, including a machine-readable medium thatreceives machine instructions as a machine-readable signal. The term“machine-readable signal” refers to any signal used to provide machineinstructions and/or data to a programmable processor.

Three aspects to HST implementation together engender its effectivenessin improving a user's vision: spatial partitioning, tailored imageprocessing, and elimination of focus ambiguity. A detailed examinationof these three benefits leads to the understanding of the advantages ofthe hardware changes that engender HST.

Spatial Partitioning

There are typically three types of viewing in an OST device,corresponding to three characteristically distinct paths for optical seethrough (OST) light rays as they travel from a viewed scene into an eyeand onto its retina. Only two types are fundamentally different, but itis convenient for the purposes of this disclosure to distinguish thethird.

Consider the drawings in FIGS. 1A-1B, which respectively depict side andfront views of one eye within an OST AR wearable platform employing ahalf-silvered mirror as an example of an optical combiner to mergeincoming scene light with the image shown on an internally-mounteddisplay. In this example, the display is mounted in the top frame of theglasses and points downward such that the mirror directs its reflectedimage into the eye; the same mirror allows light from the environment topass directly through it into the eye. Other mounting orientations andoptical combining strategies exist, including birdbath optics,electronic contact lenses, and direct laser projection, but this oneadequately illustrates the three relevant types of light paths for allof them. Skilled practitioners will be able to identify analogous pathsin all other OST platforms, regardless of architecture orimplementation-specific details.

In both FIGS. 1A-1B, labels A, B, & C represent light rays originatingin an environmental scene, directed toward the eye and travellingthrough the pupil and onto the retina. Label A indicates light thattravels directly from the scene to the retina without intersecting anymirrors, other optical combiners, or non-trivial lenses. Labels B and Cdenote light that travels from the scene and into the retina, but onlyafter passing through an optical combiner (which in this example is ahalf-silvered mirror) and/or any other incidental optical componentsupstream from the combiner.

The difference between the two is that C intersects the region of thecombiner where the internal display also projects its output. Light fromthis display does not interact with scene light at the combiner, sothere is no intrinsic difference between types B and C other than thissimple fact of geometry. However, the importance of the distinction isclarified immediately below:

Type A. For portions of the field-of-view that are not within range ofthe internal display (and also not completely or partially blocked bymirrors or refracting/attenuating optics), a direct and natural lightpath from the scene to the retina exists. This OST path cannot activelyparticipate in AR since the display cannot affect its retinal image, butits existence preserves the user's existing peripheral vision, aids inmaintaining balance and equilibrium, and maintains a fail-safe degree ofvisual capability regardless of what the internal display is showing.

Type B. For portions of the field-of-view that intersect the combiner (ahalf-silvered mirror in this example) or other optical components, butdo not overlap the projected internal display, there may be some loss ofbrightness due to attenuation as light rays of type B interact withthese optical components. Type B rays are otherwise identical to lightof type A, and can provide significant OST peripheral vision above,below, and beside the internally displayed image.

Type C. In a traditional AR application these light rays, whichintersect the projection of the internal display onto the combiner,would be blended on the retina with the image presented on the display.In hybrid see through (HST), however, this combining process that mergeslight coming from two different directions—which is the very essence ofOST AR—is deliberately prevented by the blocking type C light fromreaching the combiner with an opaque barrier so that the central visualfield of the user comprises only content originating in the display.Thus a defining paradigm is subverted, and OST eyewear locally takes oncharacteristics of video see through (VST) to form the HST architecture.

It is important to note that blocking type C rays has not been utilizedor suggested in low vision applications. OST AR displays are typicallycapable of providing light power sufficient to overwhelm the directscene image on retina, causing the brain to perceive only the dominantimage. The additional utility granted by blocking type C light with anopaque barrier will be described in a later section.

It is the partitioning of angular space into explicit OST and VSTregions that lends HST its name. The remaining two aspects serve toamplify its utility.

Tailored Image Processing

In a HST device, the image provided by the internal OST display replacesthe natural retinal image that would normally be produced by type Clight rays. The image from the display is derived in real-time from amounted camera video stream with additional processing applied.

The specific processing used with HST is described and the relevantfeatures for the present discussion include the following:

The internal display of a HST device contributes a dense replacementimage for the entire central visual field of the user, not merely anoverlay of sparse AR annotations;

Image processing is user-and-task specific, incorporating reconfigurablecombinations selected from a large palette of individual imageenhancements and modifications, but almost invariably contains someamount of magnification over at least a portion of its extent.Magnification over a portion of the field of view in an OST-styleoverlay is not viable for low-vision users due to the resultingmisalignment between foreground and background scale.

Image processing for HST devices is dynamic and responsive. For example,the above-mentioned magnification can be interactively adjusted by theuser, automatically calculated in response to the appearance of smalleror larger features (e.g. text, pictograms, or regularly structuredobjects (like a row of icons) in the camera image, or autonomouslydecided by an Artificial Intelligence (AI) or Machine Learning (ML)system that anticipates and executes user decisions by consideringcurrent conditions as well as historically observed user behavior. Ingeneral, all image modification capabilities can not only be adjustedbut also enabled and disabled at will, with response times that appearnearly instantaneous to human users. All individual processingparameters, groups of interrelated or interdependent parameters, orentire operating modes and processing chains are subject to manual orautonomous manipulation in order to tailor the processing to the user'sspecified or inferred current needs.

In a HST device, the final displayed image is adjusted to blend smoothlyinto the peripheral areas of vision of the user (formed from light raysof type A and B) where the active display does not extend. Anapproximate blend that gradually reduces the image processing so that itis minimal at the edges of the display is sufficient to be perceived asessentially ideal. This forms the unique difference between HST andother forms of AR, including VST and OST: In HST, the video imageprovided by the display comprises the central portion of the user'sfield of view. Image processing techniques described herein areconfigured to blend the displayed image in the center of the field ofview with the user's natural, unaltered peripheral vision to createenhanced vision for the user across the entire field of view.

Tailoring the central visual field to suit the user and current taskleverages a hallmark capability of the VR paradigm—absolute control overthe finest details of the retinal image—to provide flexiblecustomization and utility where it is most needed. Whereas traditionalOST AR produces displayed images that neatly coexist and integrate withthe natural scene that they overlay, HST devices, systems, and methodsmust apply carefully-selected and painstakingly-tuned nonlineardistortions to satisfy their users. Careful blending between the displayimage and the user's peripheral vision restores a naturally widefield-of-view for a perceptually-seamless user experience despite thenarrow active display region.

Examples demonstrating the use of seamless or perceptually-seamlessblending are shown in the series of FIGS. 4A-4C. FIG. 4A is anunmodified view as seen by a user wearing a HST device. Displayed image400 represents the picture displayed by the display of the HST devicewith a 50 degree (diagonal) field of view. Object 402 (here shown as atelevision remote) exists within the displayed image 400 and alsoextends beyond the displayed image 400 into the user's peripheralvision.

FIG. 4B simulates the view seen by a wearer of a HST device adjusted forseamless blending between images at the edge of the display and objectsoutside the display. The blending can be based on nonuniformmagnification via image coordinate remapping. For example, the centralpart of the display is magnified but the content as the edges of thedisplay exactly matches the original, unprocessed view (type C lightrays that are blocked) so as to align with content or objects outsidethe display region (type A and B light rays that reach the retina). Thisprovides the magnification required to counteract low visual acuity, butmaintains both image continuity and overall field of view to aid innavigation and orientation. In FIG. 4B, it can be seen that the portionof the object 402 within the displayed image 400 is magnified. However,the magnified image of object 402 is seamlessly blended at the edges ofthe displayed image with the actual object 402 as it extends beyond thedisplayed image 400 and into the user's peripheral vision.

In FIG. 4C, the requirement for blending pictures displayed by in thedisplayed image 400 with objects in the user's peripheral vision isrelaxed at the top and bottom edges, further enhancing legibilitywithout reducing field of view or the horizontal continuity that iscrucial for reading tasks. Upper and lower continuity are not asnecessary for this task, even when scanning upward or downward, so theimage mapping still appears perceptually seamless. For didacticpurposes, FIGS. 4B and 4C adequately capture the gross appearance andbasic characteristics of the total visual field imprinted onto theretina by the HST device. The static nature of the drawings belies avitally important and thoroughly novel aspect of the HST-specificcombination of actively-generated central imagery (from the internaldisplay), passively-collected peripheral imagery (type A and B light),and the tapered processing of the active display that smooths thetransition between regions.

The partitioned hybrid configuration leverages distinct characteristicsof its disparate regions to synergistically stimulate multiple aspectsof the wearer's nervous system, with results that are especiallyadvantageous to low-vision users.

A description of the complex neural- and sensory-system interplayengendered by HST partitioning, processing, and hardware follows. First,note that the HST central visual field has all of the characteristics ofa VR display. Most significantly, it is essentially disconnected fromreality by hardware-induced artifacts including camera update rate,motion blur, and latency. In contrast, the peripheral area hasabsolutely no latency associated with it. Humans are very sensitive todiscrepancies between their own motions, expectation of other motions,and delayed visual perception of said motions—in part, this leads to themotion sickness, vertigo, nausea, and general unease commonly associatedwith VR and VST AR devices. Despite the presence of such delays in theHST, however, users do not suffer from VR-induced discomfort because theneural system responsible for unconsciously maintaining a sense ofspatial orientation by reconciling proprioceptive and visual stimuliautomatically ignores the central region and locates the portion of theretinal visual field that is properly correlated with the wearer'sexpectations concerning kinematics and motion; this internal process ofextracting cues for confident equilibrium succeeds even though undelayed(type A/B) light rays only create low-resolution images on thelower-acuity peripheral areas of the retina.

Meanwhile, voluntarily-controlled parts of the brain continuouslyanalyze the higher-acuity central visual field upon which the user isconsciously concentrating and is able to recognize and accommodate thedelay and other artifacts. Low-vision users with damaged or scarredretinas naturally (or with training) learn to adjust their viewing sothat a preferred retinal locus (PRL) with relatively-better acuityreceives their focus-of-attention (and hence the central, enhanced imagefrom HST AR) so the above description still applies even with this styleof “off-center” viewing. The final aspect of HST, a gradual taper inprocessing so that the active display smoothly blends into theperiphery, is included to prevent jarring transitions from drawingconscious attention to discontinuities that might be distracting; sincethat periphery is always in a low-acuity/low-resolution portion of thetotal visual field, a perfect match is not needed andperceptual-seamlessness becomes practical with little effort. Smalldiscrepancies in alignment that arise due to user-specific variations ininter-pupil distance and facial features are also present unlessmeticulous measures are taken to eliminate them, but are likewiseessentially imperceptible.

FIGS. 5A-5B show how tailored processing is integrated into a completeHST device. FIG. 5A is a high-level block diagram focusing on data typesand dataflow, while FIG. 5B presents a flowchart that emphasizesprocesses and sequences. Together, the two give a more comprehensiveoverview because they contain both overlapping and complementaryinformation.

In FIG. 5A, raw video camera images (551) originate from a Camera (500)and enter an image-processing pipeline that performs zero or moreconfigurable modifications by Pre-transformation Enhancement blocks (502a) to produce a pre-transformation enhanced image (554A), arbitrarycoordinate transformations (suitable for uniform or non-uniformmagnification as well as blending) of the image via a Transformationblock (501) to produce a transformed and/or remapped image (553), zeroor more configurable modifications by Post-transformation Enhancementblocks (502 b) to produce a post-transformation enhanced image (554B),and optional addition of text and other annotative graphics by Overlayblocks (503) yielding the final image (555) that will be presented tothe wearer on the Display (504). Processing blocks (502 a), (502 b), and(503) are directly controlled by a collection of device modes, states,and parameters (509) that are distributed (560) to the various blocksthat depend on them.

The configuration parameters in (509) originate in a nonvolatileConfiguration Database (512) that provides persistence of user-specificpreferences and calibration data, including for example a preferredamount of edge or contrast enhancement, a required amount ofmagnification to complement the user's prescription, and controlsettings for all other features and operations. A Configurator process(511) initializes these parameters at startup, allows interactiveadjusting for calibration or tuning purposes, and manages the transferof stored or updated configuration (561) to and from the ConfigurationDatabase (512), and manages the transfer of saved or updatedconfiguration (560) to and from the active configuration parameters(509). Included among the parameters stored and manipulated by blocks509, 511, and 512 is any software-controlled occlusion information fordynamically-adjustable and customizable HST occlusion, where thespecific occlusion opacity, location, and boundary are stored andadjusted to improve viewing geometry so that it accounts foridiosyncrasies in facial structure, eye position, and relative placementof the HST device (e.g., height upon the nose) and its display.

Raw camera images (551) are also routed to zero or more optionalAutonomous Analysis/Modeling/Decision processes (505) that issueautonomously-derived control directives (558) without user input basedon image (551) contents, environmental and motion sensor data plus otheruseful data (556) from ancillary sensors or data sources (506), currentand historical device state and parameters (509, over distributionpathways 560), and current and historical direct user inputs (557)provided on interactive user controls (507). Because all autonomousprocesses (505) make decisions and issue control directivesindependently of each other and of the user, an Arbitrator (508) isnecessary to set priorities and resolve conflicts in order to produce aconsistent final set of arbitrated changes (559) to the complete set ofdevice modes, state, and parameters (509).

Unlike Enhancement (502 a, 502 b) and Overlay (503) operations, thesingle Transformation (501) block does not have control parameters thatare directly adjusted. Instead, Transformation (501) uses a quasi-staticmap (552) as a template for the transformation process. This map has thesame dimensions as the camera image (551) and comprises an implicitnon-parametric and non-procedural description of the exact remappingtransformation to be performed. It is essentially a lookup table thatuniformly accommodates standard magnification, tapered magnification forblending, and any type of deliberately controlled distortion—hence allremappings and distortions, including leaving the raw camera imageunchanged, are subjected to identical operations and computationsregardless of their complexity. The map (552) is constructed on-demandby a Map Builder (510) that recomputes its lookup table only when itscontents change, as determined by arbitrated updates (559) to relevantportions of the device mode, state, and parameter collection (509).

FIG. 5B also depicts the image processing chain, but as an orderedsequence of states. Each frame begin in state 600, captures a new rawcamera image (551) in state 601, serially applies zero or morepre-transformation enhancements (502 a) in state 603 a, applies thecurrent transformation map (552) in state 602, serially applies zero ormore post-transformation enhancements (502 b) in state 603 b, draws zeroor more overlay graphics (503) in state 604, and shows the final image(555) on the display (504) in state 605. Also performed every frame isan update (state 606) of ancillary data (556) to the most recentlyavailable values from their sources (506), followed in state 607 by anupdate of all device modes, states, and parameters (509) based on themost recently accepted changes (559) from the Arbitrator (508); thisincludes updating the HST occlusion region if it is dynamicallyadjustable and has been changed (e.g.) by the Configurator (511). Atthat point, per-frame processing concludes in state 608.

In parallel with the per-frame video processing, additional operationstake place. The autonomous processes shown as block 505 of FIG. 5A areexpanded into block 609 of FIG. 5B. Any number of these processes canexist, with each operating independently of all others. Each employsstate machines, heuristics, Artificial Intelligence, Machine Learning,or a hybrid or ad hoc approach to model and anticipate the user's needsor desires and thence autonomously control the HST device by directingchanges to its mode, state, or parameters. Every process is highlycustomized to its task, but all incorporate the high-level stepsenumerated inside block 609.

First, a decision (state 610) is made regarding whether the processneeds to analyze the most recent video frame; if not, then thatparticular process is completed (state 611). Otherwise, the autonomousprocess will analyze the image to extract features or discoverinformation (state 612), then update its internal model of the world,user, and/or device based on the image contents, ancillary information,user actions (current and historical), device state (current andhistorical), prior decisions, and any other available information (state613). It then uses this model to decide what change to the device mode,state, or parameters is needed (state 614). A decision is made based onwhether any changes are needed (state 615). If no changes are required,the autonomous process can terminate (state 616); otherwise, itsrequests will be considered by continuing to state 617.

Arbitration in state 617 is a rendezvous point that synchronizes allautonomous processes with the main image processing pipeline. When allautonomous processes have either requested changes or terminated, theArbitrator (508) will resolve any conflicts (including those resultingfrom direct user inputs, 557) and ensure that its arbitrated changes(559) will leave the device in a coherent and useful condition. State618 then determines whether the accepted changes would require an updateto the transform map (552)—if so, the map is rebuilt in state 620 basedon the applicable mode, state, and parameters (560) before computationsterminate in state 621; otherwise, termination is immediate (state 619).

In the descriptions above, it is noted that some boxes or states (viz.502 a, 502 b, 503 for FIG. 5A, plus 603 a, 603 b, and 604 for FIG. 5B)represent the serial application of zero or more operations. This is ahallmark of the layered implementation approach that allows individualenhancements or analyses to be developed independently and latercomposed in arbitrary fashion when the device is being operated. Theoutput of any such operation becomes the input of the next, withoutregard to the internal details of either. Thus, dynamic configuration tocustomize operation of the device extends beyond real-time manipulationof control parameters to include dynamic re-configuration of the entirepipeline; processing capabilities can be added or removed at any time tooptimize consumption of computing and energy resources. Application ofthese operations is partitioned into pre-transformation andpost-transformation groups (502 a vs. 502 b, and 603 a vs. 603 b) sothat arbitrary and optimized orderings of all operations includingtransformation (503, 602) can be established; although arbitraryoperations can be placed in either group, some operations such as edgeenhancement are scale-sensitive and cannot be relocated after thetransformation without detracting from performance. Thus,pre-transformation enhanced images (554 a) incorporate processing thatis order-dependent and cannot be applied after transformation (503, 602)while post-transformation enhanced images (554 b) include the totalityof all applied processing.

Functionality provided by this library of interchangeable processingmodules includes common image processing operations such as contraststretching (either static or data-dependent), contrast enhancement,color remapping (whether to ameliorate color-blindness or enhanceviewing), binarization to black-and-white, posterization to a reducedpalette of colors, and edge-detection or edge-enhancement. The commoncharacteristic of all of these types of processing is that they makeadjustments to their input image in order to produce an enhanced outputimage. Any function from commonly-available image processing or photoenhancement software running on a desktop computer is amenable to thismethodology and hence represented by boxes 502 a, 502 b, 603 a, and 603b.

The processes represented by boxes 503 and 604 can also be combined inany order (or completely omitted), but they simply add graphicaloverlays into the final output image. Each one detects classes of shapesor objects in the image in order to call attention to them. The modifiedimage does not form the input to the next process, because each processneeds to analyze the underlying image without any stray annotations.Though not explicitly shown in the figures, this means that although theintroduction of overlays is a serial process, the computations thatproduce said overlays may be parallel and in fact can base their outputson intermediate processed images from earlier processing stages,including not only outputs from the various stages of 502 a/603 a and502 b/603 b but also the original source image (551). Functionalityprovided by boxes 503 and 604 include locating and marking movingobjects in the periphery, noting changes in depth (e.g. stairs), findingsigns with readable text, highlighting signs with specific text (e.g.warnings), and detecting faces (whether known to the user or unknown).

Unlike the enhancement (502 a, 502 b or 603 a, 603 b) and overlayoperations (604), the transformation process (501, 602) is shown as asingle entity. Regardless of the complexity of the transformationselected—whether it performs simple magnification, nonlinearmagnification with blending, or leaves its input unchanged—the time andenergy cost of performing the transformation remains constant. This istrue even when multiple transformations are required, because they canbe combined and reduced to a single aggregate transformation. All maps(552) used as templates result in an identical set of computations butwith different data.

This implies that the Map Builder (510) may require complex calculationsin order to create that template. Although true, the impact is greatlytempered by the decoupled construction and application of the map. Mapcontents only require computation when the map changes, which isrelatively rare since they occur at human-driven timescales. Even then,latencies of several frames can be tolerated before becoming perceptibleto human users, allowing computations to be amortized over a long periodof time for lower peak power consumption and CPU usage.

The most straightforward implementation of the Transformation (501, 602)would compute and apply the desired remapping on a pixel-by-pixel basisfor every single displayed frame. Effectively, the entire map would berecomputed on every frame. This is a wasteful but widely-used techniquethat leverages readily-accessible hardware acceleration via the GPUresources found on typical commodity embedded systems used for AR. Forthe approach documented here, the GPU is still a viable and generallydesirable implementation target for computing maps because it allowsspecification of the map-building algorithm as a software-like processin the form of a graphics shader or compute shader using OpenGL, OpenCL,Vulkan, DirectX, SYCL, or other standard forms that provide anabstraction layer over GPU hardware resources. Here, however, thedecoupled creation and use of the template is much more efficient,reducing the likelihood of the GPU becoming temperature-throttled oroversubscribed.

The purpose of the map is twofold. Primarily, it tailors the user'sretinal image by strategically relocating and reshaping visualinformation from the camera image when projected onto the retina. Asecondary goal is to incorporate the tapered blending that integrateswith HST. A broad variety of maps are included in an HST device forlow-vision users, supporting device modes that provide configurableparameters with specific interactive behaviors and appearances. Mostusers will select a small number of modes and their accompanying mapsfrom the available gamut, using them on a regular basis to improve theirability to perform daily tasks. FIGS. 4B and 4C have already shown oneexample of the operation of an HST mapping. Following is a more generaltreatment of implementation details.

The map is a mathematical function that transforms a source FOV(captured by a camera) to a destination FOV (as presented on a display).The resulting transformation can move visual information from anarbitrary position in the source to an arbitrary location (or locations)in the destination. Mathematically, this is summarized by the equation({circumflex over (x)}, ŷ)=m(x, y); i.e. destination coordinates({circumflex over (x)}, ŷ) can be computed for any given pair of sourcecoordinates (x, y) by applying the vector function (i.e. mapping) namedm. Associated with each practically-useful m is an inverse map denotedm⁻¹ that transforms destination coordinates back to source coordinates:(x, y)=m⁻¹({circumflex over (x)}, ŷ). Ultimately m⁻¹ and m are bothsimply maps, having the same implementation; however, the relationshipis noted here since maps used for HST devices are actually inversemappings to support efficient GPU implementations which only computevalues at exact output (display) pixel coordinates.

In low-vision HST devices, these two-dimensional mappings introducecarefully controlled distortion to emphasize certain portions of theimage spatially while maintaining its overall coherence andrecognizability. They can be customized to the unique detailed needs ofindividual users, providing emphasis or enhancement to multiple areas ofany shape or location within the source visual field. While there are nofundamental limits placed on the nature of a mapping m, it is alsodesirable to provide immediate utility to a broad class of users withoutrequiring the measurement or training processes that accompanycustomization. Useful constraints that help with these goals arecontinuity and radial symmetry. Continuity avoids abrupt transitionsthat are visually jarring or distracting, and preserves theconnectedness of image contents even in the presence ofdeliberately-introduced distortion. Radially symmetric mappings arethose which, when expressed in polar coordinates (radius r and angle a)instead of Cartesian coordinates (x and y), have a simple form that doesnot depend on the angle. Hence the entire two-dimensional map is definedby the one-dimensional equation {circumflex over (r)}=m(r), where radiir and {circumflex over (r)} measure the distance from the center oftheir respective input and output images. These radially symmetricfunctions are intuitively useful even though they can only adjust themapping based on distance from the center of the image; the more generalfull two-dimensional mapping can accommodate (e.g.) off-center regions,square or elliptical regions, regions that follow user-specific contoursor boundaries of constant visual acuity, or blended combinations ofmultiple regions—without increasing implementation complexity orresource consumption. For users with a complete but localized loss ofvision due to (e.g.) retinal damage, a useful class of mappings provideswarping that pushes pixels, the center of the picture, radially outwardfrom the damaged areas, distorting the view but restoring the visibilityof obscured objects by flowing the image around the damaged retinalcells. As stated, warping can be any form of pixel or image remappingand may include, but not limited to, warping from the center or anylocation to another location, non-linear transformations, partialmagnification, blending and nonsymmetrical image reconstruction.

FIGS. 7A-7B illustrate two of the simplest radial mappings. Throughoutall sub-figures of FIGS. 7A-7B, h and w respectively denote the heightand width of the display in pixels. In FIG. 7A, the mapping is theidentity function {circumflex over (r)}=r, which results in no changesto an image because pixels at source radius r remain unmoved atdestination radius r; this corresponds to the sample processing resultillustrated in FIG. 4A. In FIG. 7B, {circumflex over (r)}=2r, resultingin pure magnification by a factor of two.

While FIG. 7A depicts a seamlessly-blended HST map, FIG. 7B does not.For perfectly seamless blending with smooth functions, it is necessarythat {circumflex over (r)}=r for r≥w/2. FIG. 7C shows how the curve inFIG. 7B can be changed to the seamlessly-blended HST map correspondingto FIG. 4B. For values of r≤0.4w, the function defining the map is astraight line with slope equal to 2.0, corresponding to uniform centralmagnification by a factor of two. For r≥w/2, the map is a straight linewith slope equal to 1.0 passing through (r=w/2, {circumflex over(r)}=w/2) as required by the above constraint. Between these two limits,there is a smooth and gradual transition (also a straight line segment,but not representing linear magnification since the extended segmentwould not pass through the origin). The entire FOV is preserved whenchanging from FIG. 4A to FIG. 4B with this mapping: even though there isuniform magnification in the center of FIG. 4B, all of the originalimage content remains visible, albeit with some distortion in theperiphery. Thus, this mapping strategically provides structural emphasis(in the form of magnification) to the part of the image that willcommand active attention and interact with the higher-acuity centralportion of the retina while still providing the full FOV as context fornavigation and orientation as well as maintaining the smooth transitionfor HST.

The specific mapping shown in FIG. 7C is an instance of the generalinverse map equation:

$r = {{m^{- 1}( \hat{r} )} = \{ \begin{matrix}\frac{\hat{r}}{g} & {{{for}\hat{r}} \leq r_{0}} \\{\frac{r_{0}}{g} + {\frac{\hat{r} - r_{0}}{r_{\max} - r_{0}} \cdot ( {r_{\max} - \frac{r_{0}}{g}} )}} & {{{for}r_{0}} \leq \hat{r} < r_{\max}} \\\hat{r} & {{{for}\hat{r}} \geq r_{\max}}\end{matrix} }$

where g is the amount of central magnification (here, g=2.0 for 2×magnification), r₀ is the radius of the centrally magnified area (here,r₀=0.4w), and r_(max)=w/2. This formulation is particularly versatilebecause it allows a small number of intuitive control parameters to beadjusted to accommodate viewing with different magnification factors,different sizes of centrally magnified areas, and different transitionregions. By setting r_(max)>w/2 it is also possible to relax theseamless-transition requirement, obtaining only perceptual-seamlessnessbut other mappings have different central, outer, and transition regioncharacteristics as evidenced in FIG. 4C. Each brings a differenttradeoff among aesthetic appearance, amount and type of distortion,computational complexity, ease of control, and other factors. Anotherexample that exhibits perfectly seamless HST transitions is given inFIG. 7D. This graph shows a continuously-varying nonlinear magnificationmap with no straight line segments except where r≥w/2—there isdistortion similar to a fisheye lens at every point within this radius.

Elimination of Focus Ambiguity

For sections of the field-of-view that coincide with the projectedinternal display (i.e. the same sections viewing the replacement image),the direct optical light path from the scene to the retina is blocked inHST. This can be accomplished by occluding the scene-facing portion ofthe optical combiner or any location in the optical processing chainthat is upstream from the point or location where the external andinternal light paths merge. Analogous locations and procedures forblocking this light can be implemented for other optical configurationsincluding those involving birdbath optics, direct laser projection, orelectronic contact lenses. It is important to note that only the portionof the combiner (or other applicable upstream, scene-facing opticalcomponent) that will directly superimpose its eye-facing output imagewith the internal display image should be blocked, because thesurrounding region can contribute significant peripheral vision.

Recall that traditional AR operations in the OST regime allow light fromthe scene to travel directly to the retina and form a natural imagethere; the internal display can then be used to overpower this naturalimage so that augmentations are visible to the user. In this presentlow-vision application, it is desirable to overpower the entire scene(within the active display limits) with an enhanced (and generallymagnified or otherwise transformed) replacement.

Typical OST AR hardware is easily capable of producing a bright enoughimage to overwhelm the natural scene image under practical lightingconditions. For users with normal vision, this is a perfectly reasonableoperating mode since they will only perceive a single dominant image,and HST as described above is viable without blocking type C light fromreaching the retina. For many low-vision users and even some otherwisenormally-sighted individuals, unfortunately, this does not hold true.

To understand why, consider the task of reading a book while using OSTglasses without any additional magnification and without blocking anydirect light path. Normal reading distance without AR gear is 16-24inches and requires accommodation of the lens within the eye to focus alegible image on the retina. The output from the internal display on ARglasses is typically collimated to appear to be originating from adistance of 8-10 feet, allowing the eyes to relax and avoid eyestrain.Without blocking the direct light path, there will be two superimposedimages formed on the OST AR user's retina—the natural image focused inthe near field, and the display image focused in the far field.

Users with normal vision can readily select between the two nearlyidentical images, shifting focus at will. In test sessions, however,low-vision users exhibited poorer reading ability even when cameraimages clearly showed increased contrast: they are unable to detect andexploit the contrast cues that normally-sighted individuals use to drivetheir focus response to completion, and hence are not able to focussuccessfully on either competing image. Problems were also noted innon-reading tasks involving intermediate-distance viewing.

FIGS. 6A and 6B (neither drawn to scale) illustrate the situationsdescribed above for a user with normal eyesight and OST AR hardware.

In FIG. 6A, the eye is shown with a relaxed lens that only focuses lightfrom relatively distant image sources onto the retina. Because of this,light emanating from the physical book in the near field—type C lightrays—is not perceived by the user. Even though the display typicallysits only a few inches from the eye (and potentially much closer withalternative technologies such as an electronic contact lens), acollimating mechanism (shown here as discrete collimating optics, thoughthey could be considered to be part of the display assembly) refract itslight into nearly parallel (paraxial) rays that appear to be emanatingfrom a virtual object much farther away. The eye in FIG. 6A sees onlythe image on the internal display, and it is in perfect focus.

In FIG. 6B, the eye is shown in an accommodated state with a more convexlens shape for near vision. This shape captures the diverging(non-parallel) type-C light rays from the book and focuses them onto theretina to form a coherent image. Paraxial rays from the display are notcoherently focused, so they are ignored. The eye in FIGS. 6A-6B seesonly the naturally-formed image of the book, and it is in perfect focus.

Most normally-sighted users can switch between the two lensconfigurations depicted by FIGS. 6A-6B at will, quickly achieving stablefocused vision even when starting from an unfocused intermediate state.This process, which is partially reflexive and partially voluntary,involves a feedback mechanism that continually adjusts the lens in adirection that produces greater contrast and clarity until asatisfactory focus is recognized.

For low-vision users, reduced visual acuity degrades contrast andclarity such that incremental changes in lens shape have less noticeableimpact on image quality. This impedes the feedback mechanism: thedirection of better focus quality is less readily determined since theimage will always remain blurred. Because any adjustment to the lens (ineither direction) always improves the focus quality of one image whilepenalizing the other; having competing images that are nearly identicalintroduces additional ambiguity. The problem persists when the twoimages are different (e.g., if the displayed image is subjected tomagnification or other processing), even when the displayed image hasbeen enhanced for easier viewing by low-acuity users.

HST solves the problem by blocking the direct light path to eliminatethe choice between competing focus distances, guaranteeing that theprocessed image is always selected. This also allows the eye to operateexclusively with a relaxed lens, avoiding eyestrain even when reading orviewing near-field objects.

Occlusion Implementation

The conventional approach to low-vision AR devices concludes thatexternal light is undesirable and must be completely blocked, resultingin the adoption of VST AR over OST AR. For users, this leads to totalimmersion that is essentially Virtual Reality with its concomitantdetrimental effects on equilibrium and mobility. By understanding theramifications of the three aspects described above, it becomes clearthat blocking only the direct external light path that coincides withthe internal display image on the retina is sufficient to alleviate theproblems associated with OST AR in low-vision applications whilepreserving peripheral vision and equilibrium. It is equally importantfor areas which do not contribute imagery from the internal display ontothe retina to remain unblocked, and even more critical that such anunblocked area be incorporated in the physical architecture of the OSTAR device with sufficient size to support the pathway for natural lightthat the human brain depends upon for equilibrium and confidentreference to the environment.

Physical implementation of HST include an appropriately-shaped physicalbarrier that prevents external scene light from impinging on areas ofthe retina that sense the internal display image; the negative space ofthis barrier implicitly enables the perceptually-seamless blending andexternal reference aspects. This physical barrier can be formed anysuitably opaque material, including plastic, vinyl, silkscreen, metal,ink, paint, dye, or even an appropriately-polarized liquid crystal. Thedegree of opacity need not be complete, as it is only necessary to biasthe user's visual system toward focusing on the internally-displayedimage in favor of the external light rays; the critical opacitythreshold for achieving this will vary across individuals, butrelatively dark occlusions that block at least 90% of external lightwill be suitable for the vast majority of users. However, the exactamount of opacity is not only dependent on the user but also dependenton the display technologies as well and includes blocking 100% and 90%of external light, but may also include blocking at least 50%, at least60%, at least 70%, and at least 80% of external light.

A simple and effective implementation, suitable for any eyeglass-likedevice with an outer lens, is shown in FIGS. 3A-3B. Here, a smallrectangle of vinyl is attached electrostatically (without adhesive) tothe non-refractive protective lens to form the HST barrier. Instead ofvinyl, alternatives include tape, paint, ink, and silkscreen or thinpieces of paper, metal, or plastic. When the inner surface of this lensis not used for optical combining, then the barrier can be located onthis surface (as in FIGS. 3A-3B) for additional isolation and protectionagainst damage or unintentional adjustments. Full-custom outer lensescan be fabricated with darkened or fully opaque sections of glass orplastic, or with opaque inserts or attachments via friction fit,magnets, adhesive, screws, heat stakes, rivets, or other commonattachment method. Since approximate blending is sufficient, barrierplacement is not critical and a slightly oversize shape can be used thatwill give satisfactory performance for a majority of users; a gradualtransition from highest opacity at the center to lower opacity near theedges also supports a broader class of users with overlapping butslightly different visual field characteristics. Some implementations(e.g., vinyl or tape) have the advantage of being adjustable.

The location of the physical barrier can vary according to the form ofthe underlying display. The barrier may be placed at any point in theoptical system prior to the location where the natural image (formedfrom type C light rays) and the internal display image are combined, aslong as the path from the internal display is not also blocked. Thisopens up possibilities for depositing a blocking material (e.g., tape,paint, ink, silkscreen, etc.) on any available pre-existing surfacewithin the AR apparatus as long as the internal display image is notimpacted, including for example the outside (scene-facing) surface of anelectronic contact lens or internal mirrors/lenses within larger ARdevices.

Instead of using an existing surface, physical barriers can also bemechanically inserted and mounted anywhere in the path of scene light aslong as display light is not also blocked. These have the potentialadvantage of being adjustable or customizable to individual users aswell as independent manufacture that gives total freedom in constructionand material choice. It is generally always possible to create a newoutermost lens on an eyeglass-like frame that provides no refraction,but serves only to provide this barrier; this even works for electroniccontact lenses, though of course it also introduces the eyeglass frame.

Included in the category of physical barriers is a blockage created byappropriately-polarized liquid crystal, as found in an LCD panel. Aconvenient implementation would embed this LCD-like structure into alens within the AR device. Then, electrical fields can be manipulatedvia electronic controls to customize the specific boundary betweentransparent and blocked regions, tuning its size and position for eachspecific user to compensate idiosyncratic facial structure and eyeposition with respect to the HST device and its display. For ease ofmanufacture or integration, an LCD panel with this degree ofcontrollability acting as the HST physical light barrier can be largerthan the required HST region—even comprising an entire an entire lens,or encompassing the user's entire field of view. These aspects apply toany medium that can be electrically polarized or manipulated to obtain atransition between states having different degrees of transparency.Electronic control gives additional flexibility since the barrier can bedeactivated, either voluntarily or automatically when power is lost (forsuperior fail-safe viewing).

The unique combination of custom processing and hardware design,including an opaque barrier, provides a HST device that leverages thebest aspects of both OST and VST AR in a single device. Where theoriginal glasses would function best in traditional overlay-based ARapplications, the addition of image processing algorithms that arecustom-tailored to magnify, improve, or otherwise manipulate thecentrally-visible image while blending seamlessly or near-imperceptiblyinto the periphery supports the enhancement needs of low-vision whilemaintaining a natural-width field of view. Working in concert with thenovelties of the adjusted hardware configuration, the processing andblending methodologies eliminate not only the equilibrium problemsassociated with VR (and VST AR) but also the focus ambiguity problemsthat low-vision users experience with OST AR. The result is a systemproviding user- and task-specific enhancements, a wide field of viewproviding the eye with precisely-controlled image details, zero-latencycues needed for confident equilibrium, and a fail-safe vision path incase of hardware failure.

While preferred embodiments of the present disclosure have been shownand described herein, for those skilled in the art such embodiments areprovided by way of example only. Numerous variations, changes, andsubstitutions will now occur to those skilled in the art withoutdeparting from the invention. It should be understood that variousalternatives to the embodiments of the invention described herein may beemployed in practicing the invention. Numerous different combinations ofembodiments described herein are possible, and such combinations areconsidered part of the present disclosure. In addition, all featuresdiscussed in connection with any one embodiment herein can be readilyadapted for use in other embodiments herein. It is intended that thefollowing claims define the scope of the invention and that methods andstructures within the scope of these claims and their equivalents becovered thereby.

When a feature or element is herein referred to as being “on” anotherfeature or element, it can be directly on the other feature or elementor intervening features and/or elements may also be present. Incontrast, when a feature or element is referred to as being “directlyon” another feature or element, there are no intervening features orelements present. It will also be understood that, when a feature orelement is referred to as being “connected”, “attached” or “coupled” toanother feature or element, it can be directly connected, attached orcoupled to the other feature or element or intervening features orelements may be present. In contrast, when a feature or element isreferred to as being “directly connected”, “directly attached” or“directly coupled” to another feature or element, there are nointervening features or elements present. Although described or shownwith respect to one embodiment, the features and elements so describedor shown can apply to other embodiments. It will also be appreciated bythose of skill in the art that references to a structure or feature thatis disposed “adjacent” another feature may have portions that overlap orunderlie the adjacent feature.

Terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention.For example, as used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising,” when used in this specification,specify the presence of stated features, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, steps, operations, elements, components, and/orgroups thereof. As used herein, the term “and/or” includes any and allcombinations of one or more of the associated listed items and may beabbreviated as “/”.

Spatially relative terms, such as “under”, “below”, “lower”, “over”,“upper” and the like, may be used herein for ease of description todescribe one element or feature's relationship to another element(s) orfeature(s) as illustrated in the figures. It will be understood that thespatially relative terms are intended to encompass differentorientations of the device in use or operation in addition to theorientation depicted in the figures. For example, if a device in thefigures is inverted, elements described as “under” or “beneath” otherelements or features would then be oriented “over” the other elements orfeatures. Thus, the exemplary term “under” can encompass both anorientation of over and under. The device may be otherwise oriented(rotated 90 degrees or at other orientations) and the spatially relativedescriptors used herein interpreted accordingly. Similarly, the terms“upwardly”, “downwardly”, “vertical”, “horizontal” and the like are usedherein for the purpose of explanation only unless specifically indicatedotherwise.

Although the terms “first” and “second” may be used herein to describevarious features/elements (including steps), these features/elementsshould not be limited by these terms, unless the context indicatesotherwise. These terms may be used to distinguish one feature/elementfrom another feature/element. Thus, a first feature/element discussedbelow could be termed a second feature/element, and similarly, a secondfeature/element discussed below could be termed a first feature/elementwithout departing from the teachings of the present invention.

Throughout this specification and the claims which follow, unless thecontext requires otherwise, the word “comprise”, and variations such as“comprises” and “comprising” means various components can be co-jointlyemployed in the methods and articles (e.g., compositions and apparatusesincluding device and methods). For example, the term “comprising” willbe understood to imply the inclusion of any stated elements or steps butnot the exclusion of any other elements or steps.

In general, any of the apparatuses and methods described herein shouldbe understood to be inclusive, but all or a sub-set of the componentsand/or steps may alternatively be exclusive, and may be expressed as“consisting of” or alternatively “consisting essentially of” the variouscomponents, steps, sub-components or sub-steps.

As used herein in the specification and claims, including as used in theexamples and unless otherwise expressly specified, all numbers may beread as if prefaced by the word “about” or “approximately,” even if theterm does not expressly appear. The phrase “about” or “approximately”may be used when describing magnitude and/or position to indicate thatthe value and/or position described is within a reasonable expectedrange of values and/or positions. For example, a numeric value may havea value that is +/−0.1% of the stated value (or range of values), +/−1%of the stated value (or range of values), +/−2% of the stated value (orrange of values), +/−5% of the stated value (or range of values), +/−10%of the stated value (or range of values), etc. Any numerical valuesgiven herein should also be understood to include about or approximatelythat value, unless the context indicates otherwise. For example, if thevalue “10” is disclosed, then “about 10” is also disclosed. Anynumerical range recited herein is intended to include all sub-rangessubsumed therein. It is also understood that when a value is disclosedthat “less than or equal to” the value, “greater than or equal to thevalue” and possible ranges between values are also disclosed, asappropriately understood by the skilled artisan. For example, if thevalue “X” is disclosed the “less than or equal to X” as well as “greaterthan or equal to X” (e.g., where X is a numerical value) is alsodisclosed. It is also understood that the throughout the application,data is provided in a number of different formats, and that this data,represents endpoints and starting points, and ranges for any combinationof the data points. For example, if a particular data point “10” and aparticular data point “15” are disclosed, it is understood that greaterthan, greater than or equal to, less than, less than or equal to, andequal to 10 and 15 are considered disclosed as well as between 10 and15. It is also understood that each unit between two particular unitsare also disclosed. For example, if 10 and 15 are disclosed, then 11,12, 13, and 14 are also disclosed.

Although various illustrative embodiments are described above, any of anumber of changes may be made to various embodiments without departingfrom the scope of the invention as described by the claims. For example,the order in which various described method steps are performed mayoften be changed in alternative embodiments, and in other alternativeembodiments one or more method steps may be skipped altogether. Optionalfeatures of various device and system embodiments may be included insome embodiments and not in others. Therefore, the foregoing descriptionis provided primarily for exemplary purposes and should not beinterpreted to limit the scope of the invention as it is set forth inthe claims.

The examples and illustrations included herein show, by way ofillustration and not of limitation, specific embodiments in which thesubject matter may be practiced. As mentioned, other embodiments may beutilized and derived there from, such that structural and logicalsubstitutions and changes may be made without departing from the scopeof this disclosure. Such embodiments of the inventive subject matter maybe referred to herein individually or collectively by the term“invention” merely for convenience and without intending to voluntarilylimit the scope of this application to any single invention or inventiveconcept, if more than one is, in fact, disclosed. Thus, althoughspecific embodiments have been illustrated and described herein, anyarrangement calculated to achieve the same purpose may be substitutedfor the specific embodiments shown. This disclosure is intended to coverany and all adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the above description.

What is claimed is:
 1. A method of providing enhanced vision for alow-vision user with a vision device, comprising: generating unprocessedreal-time video images with a camera; receiving user-specific datapertaining to at least one visual preference of the low-vision user;applying image processing to the unprocessed real-time video imagesbased on the user-specific data to produce an enhanced video stream;displaying the enhanced video stream on a display positioned within thecentral portion of the user's field of view that prevents a majority ofexternal light corresponding to the central portion of the user's fieldof view from entering the first eye of the user while allowing externallight corresponding to the peripheral portion of the user's field ofview to enter the first eye of the user, wherein displaying the enhancedvideo stream on the display allows the enhanced video stream to blendsmoothly into the peripheral portion of the user's field of view.
 2. Themethod of claim 1, wherein receiving the user-specific data comprisesreceiving an input from the low-vision user.
 3. The method of claim 2,wherein receiving the input comprises a physical interaction between thelow-vision user and the vision device.
 4. The method of claim 2, whereinreceiving the input comprises receiving spoken commands from thelow-vision user with the vision device.
 5. The method of claim 1,further comprising displaying the video stream on a second displaypositioned within the central portion of the user's field of view. 6.The method of claim 5, wherein the display is positioned in front of thefirst eye of the user and the second display is positioned in front of asecond eye of the user.
 7. The method of claim 1, wherein the at leastone visual preference comprises a magnification amount.
 8. The method ofclaim 7, wherein the magnification amount complements a visionprescription of the low-vision user.
 9. The method of claim 7, whereinthe enhanced video stream is magnified in a central portion of the videostream at the preferred magnification amount.
 10. The method of claim 9,wherein a portion of the video stream outside of the central portion ismagnified less than the central portion but more than the unprocessedreal-time video images.
 11. The method of claim 1, wherein the at leastone visual preference comprises a preferred contrast amount.
 12. Themethod of claim 1, further comprising processing the unprocessedreal-time video images to blend a top edge, a bottom edge, a left edge,and a right edge of the video stream with the peripheral portion of theuser's field of view.
 13. The method of claim 1, further comprisingprocessing the unprocessed real-time video images to blend only a leftedge and a right edge of the video stream with the peripheral portion ofthe user's field of view.
 14. The method of claim 1, further comprisingprocessing the unprocessed real-time video with image coordinateremapping to blend the video stream with the peripheral portion of theuser's field of view.
 15. The method of claim 14, wherein the imagecoordinate remapping comprises radial mapping.
 16. The method of claim1, wherein the display is positioned on or within eyeglasses of thevision device.
 17. The method of claim 1, wherein the display ispositioned on or within contact lenses of the vision device.